The Accelerated Ring Protocol: Ordered Multicast for Modern Data Centers

Ordered multicast is an important building block for many distributed applications, and there are many existing protocols providing this service, which is often referred to as total order broadcast or atomic broadcast. However, few of the existing protocols were specifically designed for high-performance in modern data center environments. This thesis introduces a new ordered multicast protocol, called the Accelerated Ring protocol, that provides high throughput and low latency in modern data center environments. Like other token-based protocols, the Accelerated Ring protocol places the protocol participants in a logical ring, and circulates a token around this ring. Each participant is allowed to send new multicasts to the other participants upon receiving the token. The key idea behind the Accelerated Ring protocol is that a participant can begin to send multicasts when it receives the token, but, unlike in other protocols, it may release the token before it finishes multicasting. Each participant updates the token to reflect all messages it will multicast during the current token round before beginning to multicast. It can then pass the token to the next participant at any point during the time it is multicasting. Since the token includes all the information the next participant needs, the next participant can begin multicasting as soon as it receives the token, even if its predecessor on the ring has not yet completed its multicasts. Sending the token before completing all multicasts allows the token to circulate the ring faster. This acceleration of token improves both throughput and latency. We implemented the protocol as a prototype and evaluated its performance in 1Gig and 10-Gig networks. We also implemented the full protocol in the open-source Spread Toolkit and evaluated its performance relative to the previous version of the toolkit in a 1-Gig local area network, virtualized 1-Gig local area settings, and a 10-Gig local area network. The results of these evaluations demonstrate the significant benefit of accelerating the token in all of these environments, both in terms of throughput and in terms of latency. The Accelerated Ring Spread implementation achieves throughput over 900 Mbps on a 1-Gigabit network and close to 2.5 Gbps on a 10-Gigabit network. At the highest throughput that both the Accelerated Ring Spread implementation and Spread 4.3 can comfortably sustain on a 1-Gig network (about 600 Mbps), the Accelerated Ring protocol reduces latency by 77% for agreed delivery and by 63% for safe delivery. On a 10-Gig network, the latency reduction is about 49% for agreed delivery and 26% for safe delivery (with throughput at 1.2 Gbps). The implementation of the protocol in Spread is open-source, and the Accelerated Ring protocol is Spread’s standard protocol for data center environments and local area networks as of Spread version 4.4.